test(e2e): add semantic live progress phases#7100
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds shared E2E phase tracking, shell instrumentation, persisted ChangesE2E runtime observability
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant LiveE2ETest
participant TestProgress
participant ShellProbe
participant E2EArtifact
participant RuntimeAudit
LiveE2ETest->>TestProgress: declare and advance semantic phases
LiveE2ETest->>ShellProbe: execute instrumented command
ShellProbe-->>TestProgress: report output and activity
TestProgress->>E2EArtifact: write test-progress.json
RuntimeAudit->>E2EArtifact: discover and validate summaries
RuntimeAudit-->>RuntimeAudit: rank p95, variability, max, and slowest phase
Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit 6432d2e in the TypeScript / code-coverage/cliThe overall coverage in commit 6432d2e in the Show a code coverage summary of the most impacted files.
Updated |
PR Review Advisor — InformationalAdvisor assessment: Informational / high confidence Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 1 optional E2E recommendation
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
|
🌿 Preview your docs: https://nvidia-preview-pr-7100.docs.buildwithfern.com/nemoclaw |
…y-hang' into codex/diagnose-agent-turn-latency-hang
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/audit-test-runtime.mts`:
- Around line 99-100: Update scripts/audit-test-runtime.mts at lines 99-100 to
use a dedicated median helper that averages the two middle sorted durations for
even-sized run sets, while preserving the existing percentile calculation for
p95. Update test/e2e/support/test-runtime-audit.test.ts at lines 56-82 to expect
the two-run median, variability, and formatted row as 30.0s, 20.0s, and 20.0s.
In `@test/e2e/live/agent-turn-latency.test.ts`:
- Around line 50-93: Remove the explicit progress.stop() call associated with
the cleanup setup around cleanup.trackDisposable. Let cleanup.runAll() execute
while progress remains active so cleanup phases and output are preserved,
relying on the progress fixture’s finally block to stop itself.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9573d819-5d22-4bb1-a36a-ae48ae47dc38
📒 Files selected for processing (17)
CONTRIBUTING.mdpackage.jsonscripts/audit-test-runtime.mtstest/e2e/README.mdtest/e2e/docs/README.mdtest/e2e/fixtures/e2e-test.tstest/e2e/fixtures/progress.tstest/e2e/fixtures/shell-probe.tstest/e2e/live/agent-turn-latency-helpers.tstest/e2e/live/agent-turn-latency.test.tstest/e2e/live/rebuild-hermes-progress.tstest/e2e/live/rebuild-hermes.test.tstest/e2e/support/agent-turn-latency-progress.test.tstest/e2e/support/e2e-fixture-context.test.tstest/e2e/support/rebuild-hermes-progress.test.tstest/e2e/support/rebuild-hermes-timing.test.tstest/e2e/support/test-runtime-audit.test.ts
💤 Files with no reviewable changes (1)
- test/e2e/live/rebuild-hermes-progress.ts
|
Fresh advisor follow-up: PRA-1 is addressed in aecce15 with a fixture-backed teardown test that verifies a completed version-1 summary, scenario, target ID, shard ID, duration, and final phase. PRA-2 does not reproduce: the live matrix sets E2E_ARTIFACT_DIR to e2e-artifacts/live (not the target directory), liveTargetTestName(target) is exactly target.id, and createArtifactSink joins those into e2e-artifacts/live/${matrix.id}/test-progress.json. The existing artifact-path test also asserts that exact root. Changing the upload contract to */test-progress.json would therefore miss the registry target artifact. |
|
The growth-guardrail failure is addressed in 950fd9c: the fixture regression now uses Vitest environment stubs and contains no added if statements. The focused fixture test and the repository test-conditionals scan both pass. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/support/e2e-progress-fixture.test.ts`:
- Around line 34-39: Consolidate the environment restoration logic in the test
fixture cleanup into a data-driven loop over the saved variable names and
values, using a single conditional to delete variables whose previous value is
undefined or restore defined values. Update the restoration block containing
previousArtifactDir, previousTargetId, and previousShardId while preserving each
variable’s existing mapping and behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b64a9b47-60f5-4b69-9f15-2584904f691d
📒 Files selected for processing (1)
test/e2e/support/e2e-progress-fixture.test.ts
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary Adds the canonical `## v0.0.92` release entry to `docs/changelog/2026-07-22.mdx` before the release plan is generated. The entry summarizes all ten pull requests merged after v0.0.91, including the OpenClaw security update and Jaeger runtime regression coverage. ## Changes - Added the canonical v0.0.92 changelog entry. - Recorded the user-visible, security, documentation, CI, and test changes in the release range. - #7280 -> `docs/changelog/2026-07-22.mdx`: OpenClaw 2026.7.1 and Node.js 22.23.1 security/runtime update. - #7378 -> `docs/changelog/2026-07-22.mdx`: canonical macOS watcher path validation. - #7379 -> `docs/changelog/2026-07-22.mdx`: stabilized full WSL platform validation. - #7380 -> `docs/changelog/2026-07-22.mdx`: bounded swap for hosted Hermes image exports. - #7100 -> `docs/changelog/2026-07-22.mdx`: semantic progress phases for live E2E tests. - #7376 -> `docs/changelog/2026-07-22.mdx`: restored v0.0.91 changelog history and corrected tagged guidance. - #7374 -> `docs/changelog/2026-07-22.mdx`: reviewed Homebrew formula transition for installer integrity checks. - #7346 -> `docs/changelog/2026-07-22.mdx`: provider-neutral headless server deployment guidance. - #7381 -> `docs/changelog/2026-07-22.mdx`: stabilized Hermes guard timing and WSL ownership fixtures. - #7339 -> `docs/changelog/2026-07-22.mdx`: real-artifact Jaeger header remediation regression coverage. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: `test/changelog-docs.test.ts` validates the canonical dated changelog and release heading contract. - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable - Station profile/scenario: Not applicable - Result: Not applicable - Supporting evidence: Not applicable ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npx vitest run test/changelog-docs.test.ts` passed 6/6 tests. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: Not applicable to a changelog-only change. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 pre-existing Fern warnings. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Carlos Villela <cvillela@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added `v0.0.92` release notes covering sandboxing updates (OpenClaw/Node.js bumps, integrity pinning remediation, mcporter handling, and upgrade validation). * Updated deployment guidance for provider-neutral headless installs, and improved live E2E test reporting plus phase-plan validation. * Tightened installer integrity-check messaging during an OpenShell Homebrew transition and expanded platform/image validation (including macOS/WSL timing) and hosted image export behavior. * Restored the previously missed `v0.0.91` changelog entry and release validation guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Carlos Villela <cvillela@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary Final-main E2E previously fanned out even when the newest applicable base-image publication was missing, partial, or failed. This change adds a trusted, fail-closed preflight that proves all required publishers succeeded before matrix generation, while preserving the controller-selected PR path without checking out PR code. ## Related Issue Fixes #7372 ## Changes - Add a dependency-free publication verifier that derives publisher inputs from the checked-in workflow, resolves the applicable first-parent commit, bounded-polls paginated GitHub evidence, validates workflow/run/job provenance across rerun attempts, and reconfirms the selected run. - Add a least-privilege `base-image-publication` job before `generate-matrix`; trusted main schedule/manual runs perform the check, controller-selected PR dispatches bypass it before checkout, and invalid invocation modes fail closed. - Add focused behavioral and workflow-boundary coverage for pagination, retries, batched pushes, merge history, partial/failed evidence, diagnostic context, controller bypass, and trust-boundary mutations. - Keep this change independent of #7100's progress artifacts and #7262's matrix timeout; preserve those adjacent lines if either PR lands first. - Scope this fix to publication sequencing. Strict digest pinning and removal of downstream mutable `latest` fallback remain follow-up work. ## Type of Change - [x] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: This changes internal GitHub Actions sequencing and provenance validation only; CLI, configuration, runtime, installation, and user-facing image behavior are unchanged. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: A final fail-closed trust-boundary review found one diagnostic-context gap, which was fixed and regression-tested; no other blocking findings remained. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed implementation - Result: `no-docs-needed` - Evidence: The change adds an automatic internal E2E publication-provenance gate without changing CLI, configuration, runtime, installation, or user-facing image behavior. - Agent: Codex Desktop - PR: #7373 <!-- docs-review-head-sha: 10a2123 --> <!-- docs-review-agents-blob-sha: 73549ba --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: `npx vitest run --project e2e-support test/e2e/support/base-image-publication.test.ts test/e2e/support/base-image-publication-workflow-boundary.test.ts test/e2e/support/e2e-workflow.test.ts` — 95 passed. - [x] Applicable broad gate passed — `npm run checks`, `npm run typecheck:cli`, and `npx prek run --from-ref origin/main --to-ref HEAD` passed. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a base-image publication verification gate that determines whether publication evidence is required before running E2E workflows. * Updated E2E matrix generation and PR E2E reporting to wait for this gate’s result. * Strengthened the E2E workflow boundary validation and approval authorization checks to rely on stable, trusted attributes. * **Tests** * Added end-to-end boundary coverage for the base-image publication policy, including classifier behavior and drift detection. * Added evidence validation tests for workflow/run selection, publisher job evidence rules, pagination safety, and deterministic polling/timeout behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Carlos Villela <cvillela@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
## Summary Follow-up to NVIDIA#7100: scheduled and ordinary manual E2E runs now publish a current-run semantic phase runtime table in the existing GitHub Actions scorecard. The table reuses the artifact audit instead of adding live-log noise, and the resume test no longer reports a bookkeeping-only phase that consistently completed in zero seconds. ## Changes - download this run's E2E artifacts in the read-only scorecard job and append the existing runtime audit table to the GitHub step summary - report the slowest observed phase's duration and `passed`, `failed`, or `skipped` outcome - keep missing or invalid progress evidence diagnostic-only so it cannot hide the main E2E result - remove `record the completed resume contract` from `onboard-resume`; the final meaningful comparison phase now owns target completion - protect the workflow boundary, native Node loading path, audit rendering, and phase plan with regression tests - document the automatic current-run table and the existing multi-run audit command ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [ ] Docs updated for user-facing behavior changes - [x] Docs not applicable — justification: no user-facing product behavior changed; the E2E maintainer guides are updated. - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: the scorecard remains read-only (`actions: read`, `contents: read`), uses an immutable artifact-action pin, treats download/audit failure as diagnostic-only, and has workflow-boundary regression coverage. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed implementation - Result: `docs-updated` - Evidence: `test/e2e/README.md` and `test/e2e/docs/README.md` document the current-run GitHub Actions scorecard table and its phase-duration and outcome fields. - Agent: Codex Desktop - PR: NVIDIA#7388 <!-- docs-review-head-sha: c190bea --> <!-- docs-review-agents-blob-sha: 560ff38 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: not applicable - Station profile/scenario: not applicable - Result: not applicable - Supporting evidence: not applicable ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — `npm exec vitest -- run --project e2e-support test/e2e/support/test-runtime-audit.test.ts test/e2e/support/e2e-scorecard.test.ts test/e2e/support/e2e-scorecard-coordinator.test.ts test/e2e/support/e2e-operations-workflow-boundary.test.ts` (61 passed); `npm run test:e2e-phases:check` (121 tests across 80 files); `npm run test:projects:check`; `npm run typecheck:cli`. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: full `e2e-support` run reached 1,343 passes; five discovery timeouts passed on isolated rerun, and one unrelated lifecycle failure reproduced unchanged in the original checkout. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - GitHub Actions scorecards now add an **E2E Test Phase Runtime** section, including a semantic phase runtime table and slowest-phase details. - Runtime reporting now includes each phase’s **duration and outcome**, with the slowest observed phase’s outcome shown as well. - **Bug Fixes** - If runtime progress artifacts are missing or invalid, scorecards render a **fallback “runtime unavailable”** message instead of failing. - **Documentation** - Updated E2E documentation to reflect the expanded runtime tables and enriched scorecard summary behavior for scheduled/manual runs. - **Tests** - Expanded E2E runtime and workflow boundary validations for the new section, artifact handling, and outcome reporting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> Signed-off-by: Carlos Villela <cvillela@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary #7100 established semantic phase coverage for all live E2E cases. This follow-up closes the remaining contract gaps without reapplying that implementation: it covers workflow-selected credential-free integration tests, requires E2E child-process boundaries to expose content-free liveness, and blocks OpenShell gateway-auth artifact uploads unless a fail-closed safety scan approves the current run attempt. Together with #7100, this completes #7101 while preserving current `main` behavior in the audited overlap areas. ## Related Issue Fixes #7101 ## Changes - Strengthen the shared progress contract with a frozen canonical capability, target/scenario identity, total and phase elapsed time, validated content-free events and activities, timestamp-only child-output observations, and final-phase enforcement. - Add `spawnObservedChild` as the audited direct asynchronous process boundary used by E2E helpers. The semantic checker rejects unaudited direct process APIs and requires synchronous calls to use a bounded timeout plus `SIGKILL`. - Add the lightweight `workflow-e2e-test` fixture for credential-free integration tests selected by the authoritative E2E planner, so they publish and validate the same semantic timeline and progress artifact contract without the stateful live fixture. - Convert the remaining agent-turn, Bedrock, Ollama, inference-routing, runtime-override, fake-server, Docker, and cleanup process paths to progress-aware boundaries without forwarding child stdout or stderr contents. - Scan final OpenShell gateway-auth artifacts before upload in both main and candidate workflows. Unsafe files are quarantined or deleted, and upload requires a run-ID/run-attempt-specific approval marker so a stale or failed scan cannot authorize publication. - Expand semantic coverage tests, workflow-boundary tests, pre-commit routing, and contributor/E2E documentation for these additional enforcement boundaries. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: An independent clean-delta and rollback audit confirmed that #7100 behavior remains intact, all 13 previously identified #7280-sensitive paths preserve current `main`, the Hermes shard implementations are unchanged, and the gateway-auth scanner/upload path fails closed. Focused scanner and workflow-boundary tests pass on the rebased commit. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## DGX Station Hardware Evidence <!-- Required only when scripts/prepare-dgx-station-host.sh changes. Maintainers must review the linked evidence before approving or merging. This is human-reviewed evidence, not authenticated hardware provenance. Exceptional bypasses use existing repository governance and must be documented on the PR. --> - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — rebased final-head E2E-support suite: 17 files and 150/150 tests passed; `npm run test:e2e-phases:check`: 123 tests across 82 files passed; selected integration suite: 6 files passed and 1 skipped, with 37 tests passed and 3 skipped; CLI and plugin type-checks, CLI build, and diff checks passed - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — local `npm run check` completed the full pre-commit stage, but its intentionally serial CLI/integration coverage lane reached the four-hour execution ceiling before producing a final summary; required sharded CI is authoritative - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added structured, redacted progress reporting across E2E phases and workflow-selected credential-free integration tests. * Introduced an OpenShell gateway auth artifact safety scan that produces an approved payload and gates evidence/artifact uploads. * **Bug Fixes** * Prevented sensitive values from appearing in progress output and persisted artifacts; uploads now proceed only after safety approval succeeds. * Hardened E2E subprocess lifecycle handling with forced termination, bounded output capture, and safer shutdown behavior. * **Documentation** * Updated E2E phase-plan/progress and artifact-safety guidance, including the `test:e2e-phases:check` contract. * **Tests** * Expanded coverage for progress reporting, observed subprocess lifecycle, and workflow/semantic-phase boundary rules. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `CONTRIBUTING.md`, `test/e2e/README.md`, `test/e2e/docs/README.md` - Agent: Codex Desktop - PR: #7397 <!-- docs-review-head-sha: bd75ae8 --> <!-- docs-review-agents-blob-sha: 560ff38 --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Carlos Villela <cvillela@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary The PR E2E gate now creates one fresh child run when the first child fails only because a standard GitHub-hosted runner disappeared. It recognizes both GitHub's canonical lost-communication annotation and the exact shutdown-log signature observed in the repeated Hermes cancellations. Assertion, timeout, OOM, policy, mixed, custom-runner, and incomplete-evidence failures remain terminal. ## Related Issue Part of #7146. Parent epic: #7140. ## Changes - Require a completed failed first child run, exact workflow identity, complete Jobs and Checks pagination, and either the canonical SHA-bound lost-runner annotation or one exact generic-cancellation annotation plus authenticated terminal shutdown evidence. - Authenticate the fallback through GitHub's job-log API without forwarding the repository token to storage. Allowlist the signed HTTPS result host, require a strong ETag and exact bounded range, and accept only the terminal runner shutdown/cleanup block bound to the cancelled step and job timestamps. Cap annotation count, per-field bytes, and normalized aggregate evidence. - Require every non-passing job to match the standard `ubuntu-latest` GitHub Actions runner, runner group, check, annotation, and terminal step-shape contract. Re-read and fingerprint all run, check, annotation, and optional log evidence before dispatch. - Create a fresh child run and replacement required check for the one retry. Preserve the first run, check, state, plan, and evidence as linked diagnostic history. - Revalidate the PR SHA, base SHA, workflow SHA, risk plan, state, and retry history before each privileged mutation. - Keep coordination checks immutable across base identities so a controller that resumes after a rapid retarget cannot fail the newer base's gate. - Terminalize the second attempt and clean partial replacement state after failure, cancellation, skipped setup, interruption, or response loss. The required observer follows the latest exact check. - Serialize gate mutations by repository, PR number, PR SHA, and base SHA. Keep up to 100 pending exact-identity mutations in GitHub's FIFO queue without canceling their owner. - Bound each child wait at 140 minutes, the controller at 330 minutes, the observer at 358 minutes, and the required job at 360 minutes. - Document the strict retry criteria, lineage, cleanup behavior, and timeout envelope in `test/e2e/README.md`. - Keep merged #7100 progress telemetry and #7399 resource telemetry unchanged. This PR does not add a second sampler or retry final-main runs. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: an independent 9-category audit passed at `697d78e5f`; it verified annotation and log authentication, token isolation, bounded transport, spoof rejection, immutable retry lineage, two-pass evidence revalidation, cross-base check ownership, cleanup, timeout arithmetic, and FIFO mutation ownership with no unresolved finding. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `test/e2e/README.md` - Agent: Codex Desktop - PR: #7423 <!-- docs-review-head-sha: d1f8247 --> <!-- docs-review-agents-blob-sha: 560ff38 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable - Station profile/scenario: Not applicable - Result: Not applicable - Supporting evidence: Not applicable ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 14 PR-gate integration files passed 244 tests at `697d78e5f`; the focused classifier/transport suites passed 72 tests; the test-only review fix passed all 26 controller tests at `ac430dcf7`; the documented fallback lifecycle passed docs validation and all 39 classifier tests at `d1f824728`; CLI and JavaScript type checking, changed-file hooks, source-shape, test-size, Biome, YAML, Markdown, gitleaks, and diff checks passed. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — targeted controller coverage is authoritative; GitHub CI will run the repository gate. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — passed with 0 errors and 2 existing Fern warnings - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) — no `docs/` page changed - [ ] New doc pages include SPDX header and frontmatter (new pages only) — no new doc page --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a one-time automated PR E2E retry for hosted-runner loss, including dedicated retry and abandon lifecycle steps with evidence download/verification. * Updated PR E2E gate coordination to run retry-sensitive paths only on the first workflow attempt using attempt-aware, per-PR/per-SHA concurrency isolation. * **Bug Fixes** * Hardened gate/approval execution to prevent cross-run collisions and incorrect retry/authorization, with improved fail-closed behavior. * Increased relevant time budgets and tightened “superseded/cancel” handling. * **Documentation** * Expanded PR E2E gate flow documentation for retries, evidence rules, and terminal outcomes. * **Tests** * Expanded end-to-end coverage for runner-loss classification, retry lifecycle, authorization, and command parsing edge cases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
## Summary Records bounded start/end CPU, memory, and workspace evidence for every #7145 larger-runner route so the standard-versus-larger experiment can compare like-for-like jobs. This is the current-`main` replacement for the unique telemetry portion of closed #7247; it does not change runner routing or duplicate #7100 phase, stall, timing, or resource diagnostics. ## Related Issue Related to #7145. Supersedes the telemetry portion of #7247. ## Changes - Write one canonical private numeric sample after workspace preparation and one from an `always()` finalizer before artifact scanning or upload. - Cover 12 routed workflow lane identities and 13 concrete executions; `hermes-inference-switch` contributes its `hosted` and `anthropic` matrix entries. - Keep the OpenClaw entries for `mcp-bridge`, `channels-stop-start`, and `security-posture` uninstrumented because #7391 leaves those entries on standard runners. - Summarize measured-window CPU utilization, host-memory endpoints and root-cgroup lifetime peak, plus workspace endpoints and growth without invoking Docker, `ps`, or background samplers. - Reject unknown fields, duplicate JSON keys, malformed timestamps, identity/counter drift, oversized evidence, symlink/hardlink final-file substitution, matrix topology drift, and workflow scope/order drift. - Document comparison controls and limitations. Standard-runner baselines can begin before #7391 changes routing; the same evidence remains after larger-runner activation. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: independent review confirmed the exact trusted-main/no-alternate-checkout boundary, Hermes-only matrix guards, bounded private schema, and finalizer-before-scan/upload ordering. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed implementation - Result: `docs-updated` - Evidence: `test/e2e/README.md` documents the 12 routed workflow lane identities, 13 concrete executions, OpenClaw exclusions, sample limitations, missing-summary interpretation, and comparison requirements. The exact-head refresh confirmed the final test-only review fix requires no further documentation change. - Agent: Codex Desktop - PR: #7399 <!-- docs-review-head-sha: c09137d --> <!-- docs-review-agents-blob-sha: 9061453 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: Not applicable - Station profile/scenario: Not applicable - Result: Not applicable - Supporting evidence: Not applicable ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — focused E2E-support validation passed 102 tests; an independent rerun of the four directly affected suites passed 94 tests; the final matrix-assertion review fix passed its focused 28-test suite. - [ ] Applicable broad gate passed — Full `e2e-support` previously reached 1,381 passed and 9 skipped; two unrelated existing local failures remained. GitHub CI is authoritative for the Linux gate. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — completed with 0 errors and 2 existing Fern warnings - [ ] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) — no `docs/` page changed - [ ] New doc pages include SPDX header and frontmatter (new pages only) — no new doc page --- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
<!-- markdownlint-disable MD041 --> ## Summary This adds bounded periodic and phase-aware resource telemetry to the canonical runner-comparison ledger for trusted heavy E2E executions. It reuses the existing progress pulse and records only numeric or fixed-enum evidence, improving runner-pressure diagnosis without changing failure classification or retry policy. ## Related Issue Related to #7145 and #7146 (parent epic: #7140). ## Changes - Extend the canonical runner-comparison artifact to a bounded v2 lifecycle with private `0600` files, a reserved final sample, v1 read compatibility, and deterministic summaries. - Capture bounded host, cgroup, PSI, filesystem, Docker, container, and coarse process-class evidence at endpoints, phase boundaries, and an approximately 60-second cadence. - Reuse the existing progress timer with fixed deadlines, no catch-up bursts, per-scenario attribution boundaries, and a permanent fallback to the existing bounded collector if canonical sampling fails. - Add focused privacy, compatibility, timing, fallback, aggregation, summary, and workflow-boundary coverage, including the three-shard `common-egress-agent` topology merged in #7448. - Document the telemetry lifecycle, probe budgets, privacy limits, compatibility behavior, and the separation from classification and retry. This builds on the merged progress/timing work from #7100 and #7213 and extends the canonical ledger from #7399 rather than creating parallel infrastructure. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates <!-- Check one tests line and one docs line. Check other lines when applicable. Add every requested justification or approval reference. --> - [x] Tests added or updated for changed behavior - [ ] Existing tests cover changed behavior — justification: - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [x] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [x] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent exact-head privacy, schema, classifier, retry, and compatibility review at `419182d89` found no actionable issues. - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review <!-- Required for code and documentation changes after the changes and applicable validation are complete. Keep one review checkbox and one instance of each visible or hidden field. For Evidence, list changed documentation paths. For documentation-only changes, also state that the writing rules and documentation style were reviewed. For other results, explain why no documentation change is needed or why the review is blocked. For Agent, use a consistent product and surface name, such as Codex Desktop, Codex CLI, Claude Code, or Cursor. After committing all review changes, put `git rev-parse --short HEAD` and `git rev-parse --short HEAD:AGENTS.md` in the hidden metadata below. Rerun the review and refresh that metadata after any new commit. This receipt is advisory during the data-collection pilot. --> - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `test/e2e/README.md` - Agent: Codex Desktop <!-- docs-review-head-sha: e06653c --> <!-- docs-review-agents-blob-sha: 9c9b36d --> ## DGX Station Hardware Evidence <!-- Required only when scripts/prepare-dgx-station-host.sh changes. Maintainers must review the linked evidence before approving or merging. This is human-reviewed evidence, not authenticated hardware provenance. Exceptional bypasses use existing repository governance and must be documented on the PR. --> - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification <!-- Check each applicable item only when supported by the requested evidence. Run targeted tests once per relevant change set and rerun after later edits or hook autofixes that can affect the tested behavior. Do not rerun hook-covered checks. --> - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run check:diff` passed when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: Exact-head serialized telemetry/progress/workflow suite passed 225/225 tests across 15 files; after the linear-test guardrail repair, the primary focused suite passed 165/165 across 9 files, Vitest membership was exact across 1,764 files, CLI typecheck passed, and `npm run check:diff` passed. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: A broad E2E-support run exposed only parallel-load timeouts and one macOS lifecycle failure reproduced unchanged on `origin/main`; scoped exact-head gates are listed above. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- <!-- DCO sign-off is required in this PR description, and every commit must appear as Verified in GitHub. Run: git config user.name && git config user.email --> Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added runner-comparison schema v2 telemetry as a bounded, ordered ledger with periodic, scenario-start, phase, and finalize samples. * Enabled live E2E runner-comparison sampling and added conditional periodic sampling in the E2E progress harness. * Introduced pulse-based progress resource sampling with configurable interval and best-effort phase/scenario capture. * **Bug Fixes / Improvements** * Switched memory diagnostics to available-memory reporting (with free-memory fallback labeling). * Hardened telemetry parsing/validation, evidence sanitization, and expanded runner-pressure/resource diagnostics. * **Documentation** * Documented v2 ordering/cadence, privacy/size limits, and diagnostic-only summary guidance. * **Tests** * Added/expanded E2E tests for v2 schema validation, timing/cadence, fallback behavior, and updated expected outputs. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Apurv Kumaria <akumaria@nvidia.com> Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com> Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Summary
Live E2E jobs now report the semantic phase they are executing without repeating Vitest's test-level start, identity, elapsed time, or completion output. Every live test declares an individualized ordered plan, and each completed phase records its outcome and duration so a stalled or failed matrix job shows where time was spent.
Related Issue
Part of #7101
Changes
passed,failed, orskippedoutcomes and per-phase durations, then append a harness-owned resource-release phase so cleanup hangs and failures are timed independently.test-progress.json, including the dedicatedskill-agentjob upload.Type of Change
Quality Gates
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpm run typecheck:cli, project membership, source-shape, file-size, changed-file Biome, andgit diff --checkpassednpm testfor broad runtime/test-harness changes;npm run checkfor repo-wide validation/coverage changes — command/result: required CI is authoritative for the broad sharded suitenpm run docsbuilds without warnings (doc changes only) — validation passed; Fern reported 2 existing warningsSigned-off-by: Charan Jagwani cjagwani@nvidia.com
Summary by CodeRabbit
New Features
test-progress.jsonartifact (including target/shard metadata when available).Documentation
Tests